Auto-calibration of multi-proiector systems 



Introduction 

The invention relates to a method for the automatic calibration of a multi- 
projector system, which allows to superimpose any number of images 
produced by different projectors with exact congruence of the pixels in order 
to increase the total luminosity or to facilitate a stereoscopic representation 
or to represent an image of several projection images overlapping accurately 
to the pixel on a (big) screen. 

Description of the problems 

In order to increase the total luminosity of an image or to produce a passive 
stereoscopic image, it is necessary that several projectors illuminate the 
same area of the projection surface. With this image representation, there is 
the problem that the single images of the projection units normally are not 
superimposed. Presently, this calibration is made by hand and involves great 
efforts and is too imprecise. 

For a long time, there has moreover been the desire to create a possibility to 
increase the pixel resolution of a projection beside the geometry rectification 
and the increase in luminosity. Since projectors with high resolutions (> 
1024 x 768) are very expensive, the obvious solution was to achieve this 
increase via several projectors. Typically, the associated problem was that it 
is impossible to position the projectors adjacent to each other without 
noticeable transition. Mostly, the projectors are oriented obliquely with 
respect to each other and have a brighter overlapping area. 

These effects have to be compensated by means of a system calibration. 
The present state of the art is predominated by inflexible expensive manual 
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calibration. There are, however, first attempts at an automatic calibration 
which, however, are less robust. 

It is aimed at implementing an auto-calibration with exact congruence of the 
pixels which makes the troublesome manual fine adjustment superfluous. By 
this calibration, the projection surfaces are oriented to each other, di- 
stortions by the geometry of the projection surface are compensated for and 
the individual projectors are blended into each other such that the 
overlapping area is no longer perceptible and a projection consisting of n x 
m projectors appears as a single-projector display to the user. 

US-B-6,222,593 describes a multi-projector system where the projection 
image consists of several overlapping partial images each of which is pro- 
jected by a different projector, the total projection image being picked up by 
a camera. The control of the projectors is effected in dependence on the 
respective distance and the respective orientation (rotation about three 
angles in space) of the projectors relative to the projection surface. Because 
of this merely global image transformation, the known system cannot be 
used with arbitrarily shaped projection surfaces. 

Further multi-projection systems are described, for example, in WO-A- 
00/18139 and WO-A-00/07376 as well. Here, a calibration of the system is 
effected as well, but it is always assumed that the projection images are 
distortion-free, i.e., that the projection surface is plane and particularly 
vertical to the optical axes of the projectors and a camera. 

It is an object of the invention to implement an auto-calibration with exact 
congruence of the pixels, which makes the troublesome manual fine 
adjustment superfluous. Additionally, this calibration also compensates for 
image distortions (with respect to the camera) caused by the geometry of 
the projection surface and the projector (Keystone), respectively. 
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In order to solve the afore-mentioned object, the invention suggests a 
method for the automatic calibration of a multi-projector system provided 
with at least two projectors for projecting images onto a projection surface, 
a digital camera for picking up the projection surface, and a control unit for 
controlling the projectors and the camera. 

The auto-calibration is performed according to the following steps: 

- projecting patterns of individual single strips of intersecting lines parallel 
in groups onto the projection surface by each of the projectors, the 
positions of the line intersection points of the single-strip patterns in the 
image to be projected being known, 

- capturing the projected single-strip patterns by means of the camera, 

- filtering the captured single-strip patterns for compensating for back- 
ground noise, ambient light and/or optical distortions of the single-strip 
patterns caused by the camera and for detecting the line intersection 
points of the captured single-strip patterns, 

- detecting the largest projection surface possible on the basis of the 
captured single-strip patterns, 

- comparing the positions of the line intersection points within the captured 
single-strip patterns with the known position of the line intersection points 
of the single-strip patterns to be projected by the projectors in order to 
detect correction data for correcting distortions occurring as a 
consequence of unevennesses of the projection surface, 

- calculating warp fields and image warping corresponding to the cor- 
rection data. 

In case of applying the invention to a projection system where the projection 
images of two (or more) projectors are superimposed as extensively as 
possible (e.g., for increasing the total luminosity of the projected image or 
for the stereoscopic representation), the largest projection surface possible 
is chosen as the largest common rectangular surface possible within the 
overlapping area of the projection surfaces of all the projectors. On this 
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occasion, the projectors project the same image information onto the 
projection surface showing the superimposed projection images. Thereby, 
the luminosity on the projection surface is increased. 

If, however, a projection system is concerned where the individual areas of 
a total projection image are produced by projectors projecting partial 
images, the individual partial projection images overlap each other. In this 
case, the maximum projection surface is that rectangular surface comprising 
all partial projection surfaces. 

In the invention, standard PC technology, i.e., no expensive special hard- 
ware, is used. The projectors used (also referred to as beamers) are 
computer-controlled and available on the market. 

The filtering of the strip patterns captured by the camera or by a ccmmon 
optical capturing device serves the image processingand compensates for 
distortions or other optical mistakes/shortcomings of the optical capturing 
device. When the lines of the strip patters have a width of, e.g., one pixel, 
that does unfortunately not mean that the lines captured by the capturing 
device are only one pixel wide; these lines rather have a width of several 
pixels. But in order to be able to compare the position of the intersection 
points of the projected lines with those of the intersection points of the 
captured projected lines, the intersection points (x/y coordinate of a pixel) 
have to be detected from the Aintersection fietdse by averaging. Besides, it 
has to be considered that the edges of the lines are serrated in the 
photograph when the projected lines do not exactly correspond with the row 
and column alignment of the pixels of the camera, which will be the normal 
case. 

Hereinafter, the partial steps of the auto-calibration shall be explained with 
respect to the Figures. 



In the Figures: 
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Fig. 1 shows an experimental set-up with two projectors according to a 
first embodiment of the invention, 

Fig. 2 shows a background image (projection of two black surfaces), 

Fig. 3 shows a horizontal strip projection of the left projector of Fig. 1, 

Fig. 4 shows a horizontal strip projection of the right projector of Fig. 1, 

Fig. 5 shows a vertical strip projection of the left projector of Fig. 1, 

Fig. 6 shows a vertical strip projection of the right projector of Fig. 1, 

Fig. 7 shows a detected grid network and the largest common projection 
surface of the experimental set-up according to Fig. 1, 

Fig. 8 shows a warped image of the left projector of Fig. 1, 

Fig. 9 shows a warped image of the right projector of Fig. 2, 

Fig. 10 shows a rectified image of the left projector (the right projector is 
covered) of Fig. 1, 

Fig. 11 shows a rectified image of both projectors of the experimental set- 
up of Fig. 1 in superposition, 

Fig. 12 shows a comparison of the individual luminosity of the two pro- 
jection images of the experimental set-up according to Fig. 1, 

Fig. 13 shows a distorted image of a 2 x 2 projector cluster according to a 
second embodiment of the invention, 
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Fig. 14 shows the largest projection surface in the 2x2 projector cluster 
according to Fig. 13, 

Fig. 15 shows the rectified total image (without blending) of the 2x2 
projector cluster according to Fig. 13, 

Fig. 16 shows the rectified total image (with blending) of the 2x2 projector 
cluster according to Fig. 13, and 

Fig. 17 shows partial results of the algorithms which are shown by way of 
example for a partial projection. 

Embodiment 1 

Hereinafter, a first embodiment of the invention will be explained with 
reference to Figs. 1 to 12, wherein the projection images of two projectors 
are superimposed nearly completely to produce stereoscopic images or 
projection images of increased total luminosity. 
Producing, capturing and image filtering of the strip patterns 

To be able to perform a successful geometry rectification by means of image 
warping, the basic geometry of the projection surfaces has to be detected 
first. This is done by capturing projected vertical and horizontal strip 
patterns. These strips have a constant distance which has to be selected in 
dependence on the complexity of the geometry to be rectified. In the 
embodiment described herein, a strip distance of 20 pixels has been selected 
since the wall had many irregularities. Different strip distances and 
arrangements are possible. From the courses of the lines (curved, straight, 
oblique) and the line interspaces, the geometry of the projection surface can 
be inferred. For all projectors, the projected strips are captured by a digital 
camera (in our case, two projectors meaning two vertical and two horizontal 
strip images). Additionally, an image is captured where both projectors are 
switched to black to be able to remove the partly disturbing background and 
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low noise, respectively, from the camera image. Before the image filtering 
continues, this background image is subtracted from all captured images. 
The captured images can be seen in Figs. 2 to 6. 

Due to the use of a lens camera, the captured images have typical radial 
distortions which have to be removed before the further processing. In case 
of a distortion-free capturing, this step can be omitted. 

The actual filtering has been effected by standard image processing filter 
operations. After the image subtraction of image to be filtered and back- 
ground image, the resulting image is transformed into shades of gray so 
that each of the three color channels has the same color value. Subse- 
quently, this image is transformed into a black/white image in dependence 
on a threshold value, the strips being white and the background being black. 
A dilation and erosion is performed to remove holes in the strips. 
Subsequently, the lines still have to be filtered such that they are only one 
pixel wide. This is effected by means of a vertical and horizontal offset filter, 
respectively. This offset filter is performed in two directions (left and right 
and up and down, respectively). The average of the results of these two 
filtering operations yields the center of the strip. Now, the filtered strips are 
represented by their centers. 

All filtering operations have been performed by means of pixel shaders 
known per se, with hardware assistance. 

Calculation of the largest projection surface possible 

To guarantee a convergent perception, the largest common projection 
surface of all projectors has to be determined in the original image ratio (4:3 
in this embodiment). To this end, a grid network is produced from the 
filtered strip patterns first. By means of an optimizing algorithm, the largest 
common projection surface is Alooked for@ within this network which, in our 
case, consists of two partial grid networks. The grid of the left and right 
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projectors as well as the target projection surface are illustrated in Fig. 7. 
From these data, the parameters for the warping can be determined 
subsequently. 

Calculation of the warp fields and image warping 

By means of the afore-calculated grid, the image can be split into warping 
patches. These patches consist of a dest patch (the image area into which 
an image section is to be warped) and a src patch (the image area warped 
into the dest patch). 

The calculation of the corner points of the dest patches is trivial. Thus, the 
position of all the grid points in the target is known as well due to the given 
distance of the horizontal and vertical lines. 

The src patches are calculated linearly in dependence on the projection 
rectangle (in the photo) and the position of the dest patches (in the photo). 
The calculation for the left lower corner of a src patch in dependence on the 
left lower corner of a dest patch is indicated: 

patch. xl = (pLB.X - projectionRect.left) / (projectionRect.right - 
projection Rect. left); 

patch. yl = (pLB.Y - projectionRect. bottom) / (projectionRect.top - 
projection Rect. bottom ). 

The src patch is transformed into the dest patch by means of a warping 
process including integrated perspective correction. 

If the distance of the horizontal and vertical lines is selected to be relatively 
small, it is also possible to realize the compensation of curvatures. It has 
already been possible to compensate for curvatures of the wall with the line 
distances of twenty pixels selected in the experiments so that the projected 
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image on the curved projection surface appears to the viewer (from the 
camera) as being projected onto a plane surface. 

In case of a calibration of two projectors, two warp arrays have to be 
calculated, in case of more than two projectors correspondingly more. The 
warp arrays are always determined in dependence on the corresponding grid 
points, with the restriction of the common largest projection surface. The 
warped images calculated for a two-channel projection can be seen in Figs. 8 
and 9. The result at the projection surface for one projector can be seen in 
Fig. 10 and for two projectors in Fig. 11. The comparison of the luminosity is 
illustrated in Fig. 12. 

To ensure an effective processing, the auto-calibration is made only once 
(per session). For this reason, the data calculated for the geometry recti- 
fication and multi-projector calibration are stored in warp fields which can be 
used by the X-Rooms™ system (K. Isakovic, T. Dudziak, K. Kochy. X- 
Rooms. A PC-based immersive visualization environment, Web 3D Confe- 
rence, Tempe, Arizona, 2002). The rectification by means of the warp fields 
has been done by hardware assistance again so that a rectification in real 
time is possible. 

Application fields 

There are many application fields for a multi-projector calibration. It can be 
used, for example, for a cost-effective luminosity increase or for stereoscopic 
calibration. Furthermore, a rectification necessitated by the geometry and 
the projector can be performed without any great problems. Experiments in 
practice have shown that the auto-calibration of a two-projector system by 
means of the method according to the invention takes only about 35 
seconds including the image capturing (approximately 20 seconds). Thus, 
the invention represents a significant increase with respect to the manual 
calibration. 
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Embodiment 2 

In the following text, an extension of the afore-described method for the 
auto-calibration of multi-projector systems is presented with respect to Figs. 
13 to 17. While the possibility of superposing any number of images with 
exact congruence of the pixels has been described in detail above, the ability 
of the system of automatically calibrating projector clusters present as n x m 
projector matrix will be described in the present text. 

For a long time, there has existed the desire of an increase in the pixel 
resolution of a projection beside the geometry rectification and the increase 
in luminosity. Since projectors with high resolutions (> 1024 x 768) are very 
expensive, the obvious solution was to achieve this increase via several 
projectors. Typically, the associated problem was that it is impossible to 
position the projectors adjacent to each other without noticeable transition. 
Mostly, the projectors are oriented obliquely with respect to each other and 
have a brighter overlapping area (see Fig. 13). 

These effects have to be compensated for by a system calibration. The 
present state of the art is predominated by inflexible expensive manual 
calibration. There are, however, first attempts at an automatic calibration 
which, however, are less robust. 

It is aimed at implementing an auto-calibration with exact congruence of the 
pixels which makes the troublesome manual fine adjustment superfluous. By 
this calibration, the projection surfaces are oriented to each other, di- 
stortions by the geometry of the projection surface are compensated for and 
the individual projectors are blended into each other such that the 
overlapping area is no longer perceptible and a projection consisting of n x 
m projectors appears as a single-projector-display to the user. 

Pr ducing, capturing and image-filtering of the strip patterns 
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This part of the method has already been described above. Therefore, it is 
not intended to go into further details here. 

Calculation of the largest projection surface possible 

To guarantee a convergent perception, the largest common projection 
surface of all projectors has to be determined in the original image ratio. 
This image ratio is determined from the number of projectors and the 
arrangement thereof. A cluster with 3 projectors in horizontal direction, for 
example, has an image ratio of 12:3, the 2x2 system presented in detail 
herein a ratio of 4:3. 

From the strip patterns filtered before, a grid network is produced again. 
The largest overlapping projection surface found automatically by the 
optimization algorithm as well as the grid network on which it is based are 
illustrated in Fig. 14. 

Calculation of the warp fields and image warping 

The matrices used for the warping of the partial images are calculated 
according to the above-described method again. The rectified total image 
(still without blending of the overlapping areas) is shown in Fig. 15. 

Blending of the overlapping projectors 

The total image produced by the image warping appears already geome- 
trically homogeneous, but has a greater luminosity at the overlapping areas. 
Therefore, it can still be identified as a composed image very easily. It is the 
aim of the blending to change this overlapping area such that the image 
appears homogeneous as to color as well. 

During the pre-processing, a blended image in the color white is produced 
for each projector. This image has the total resolution (i.e., 2048 x 1536 
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pixels in case of 2 x 2 projectors) and is warped into an image with projector 
resolution (1024 x 768) after the blending values have been calculated, and 
subsequently, it is used as a masking texture (multiplication of the already 
warped image in current graphic hardware per multi-texturing). 

In the first step for the calculation of the blending values, the grid data of 
the individual projectors are arranged in a n x m matrix so that they are 
sorted according to their positions. In the next step, n projectors (2 in our 
case) are blended with each other horizontally. To this end, a lower and an 
upper limit is determined across the projection rectangle as well as all the 
projector grid data (per horizontal strip). These limits lie within the 
projection surface and within the n projection surfaces to be blended and 
further, they are parallel to the projection surface. This limit simplifies the 
blending to be a simple scan line process. 

The horizontal scan line blending is always effected between 2 overlapping 
images. The right edge is represented by the right edge of the left partial 
image, the left edge by the left edge of the right partial image. Per line, a 
linear color value decrease is effected from the starting to the final point. 
The color values already listed in the blending image - so that a multiple 
blending is possible as well - and a multiplication factor determined per pixel 
of the scan line (over the entire scan line of 1.0 60.0) serve as a basis for 
this color value decrease. Subsequently, the linear blending values 
determined in this manner are provided with a gamma correction 
(MnBlend 10/gamma ). This gamma value depends on the projector and can be 
indicated for each color channel (RGB). The definitely calculated blending 
value is stored both in the left and the right blending image - depending on 
the starting and final points of the scan line. Due to the different start and 
final values (the starting point in the left partial image corresponds to the 
final value in the right partial image), a so-called cross fade with integrated 
gamma correction is created. 
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Subsequently, all values of the blending image below the lower limit and 
above the upper limit, respectively, are blackened. 

After the horizontal blending of all strips (2 in our case), a vertical blending 
still has to be performed. This is also done by the scan line process, here, 
however, the starting value and final value per vertical scan line are defined 
by the upper limit of an already horizontally blended strip and the lower limit 
of the strip arranged above and already blended horizontally. 

The blending images in total projection size now available are transformed 
into n x m blending images in projector resolution by means of image 
warping and the already available warp arrays. Now, these blending images 
can be used in real time for the blending of multi-projector systems by 
means of multi-texturing. 

The result of the blending of a 2 x 2 projector cluster can be seen in Fig. 16. 
By way of example, Fig. 17 shows a surview of all operational steps at a 
partial image: 

- distorted partial image on the projection surface (upper line, left picture), 

- pre-warped partial image (middle line, left picture), 

- associated blending image (upper line, right picture) 

- pre-warped partial image multiplied with the blending image (middle line, 
left picture), 

- warped, non-blended partial image on the projection surface (lower line, 
left picture), 

- warped blended partial image on the projection surface (lower line, right 
picture). 

Application fields 

The automatic multi-projector calibration presented herein can be used 
anyplace where large-scaled projections of high quality are to be produced 
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at low costs and without consuming much time (fairs, showrooms, HD TV, 
digital cinema). With the calibration presented herein, a geometry 
rectification as well as an image stitching and an image blending are 
performed. With the method presented herein, the auto-calibration of a 2 x 
2 cluster takes about 1 minute including the image capturing (about 30 
seconds) with standard PC components. Thus, it represents a significant 
increase over the manual calibration. 



